Content Based Retrieval of VRML Objects - An Iterative and Interactive Approach

We examine the problem of searching a database of three dimensional objects given in VRML for objects similar to a given ob ject We introduce an algorithm which is both iterative and interactive Rather than base the search solely on geometric feature similarity we propose letting the user in uence future search results by marking some of the results of the current search as relevant or irrelevant thus in dicating personal preferences A novel approach based on SVM is used for the adaptation of the distance measure consistently with these mark ings which brings the relevant objects closer and pushes the irrelevant objects farther We show that in practice very few iterations are needed for the system to converge well on what the user had in mind Introduction The problem of automatically comparing objects and detecting which ones are alike is a di cult problem After all similarity is in the eye of the beholder Objects similarity is a subjective matter dependent on the human viewer since objects have semantics and are not only geometric entities Does a parasaurolo phus Figure look more like a kangaroo or more like an allosaurus A Parasaurolophus A Kangaroo An Allosaurus Fig Which one is more alike Finding similarity between geometric objects has been a lively topic of re search in computational geometry Objects are considered similar if their geometric features are close given a metric measuring the distance A lot of work has been done matching sets of points in two dimensions under var ious transformation sets The Hausdor distance has been frequently used to measure the distance between point sets Polygonal shapes have also been considered Algorithms for computing the minimum Hausdor distance between polygons were proposed by Hausdor distance has some limitations as it is not robust for outliers Compar ing polygons as turning functions is proposed in In the Frechet distance is used to compare polygons In a re ection metric is de ned and computed for two unions of line segments Less work has been invested in the three dimensional case In particular extending methods of comparing polygonal curves in two dimensions to higher dimensions is non trivial both in theory and in practice However as VRML objects are becoming more popular on the World Wide Web this problem is expected to have many applications one prominent example being in e commerce Much of the research on similarity has been done in the context of images On one hand nding similarities between three dimensional models seems like it should be easier than nding similarities between their projections to images After all the whole object can be seen thus occlusions self occlusions lighting e ects and re ections are avoided On the other hand three dimensional models can be harder to handle since they do not have a simple parameterization and registration and feature correspondence are more di cult to nd In this paper we focus on nding similarities between three dimensional ge ometric objects Rather than dealing solely with the choice of features and the de nition of a distance measure on these features we want to give the user the added ability of in uencing the search as it is being conducted by means of rele vance feedback We suggest an iterative method where each user can specify how relevant the results are according to this user s preferences The scheme we propose is as follows Given an object the system searches a database for similar ones Once a set of results is obtained the user is presented with the best most similar objects and is given a chance to mark a subset of them as relevant or as irrelevant Using this feedback from the user the dis tance function is updated and another iteration of the search may be conducted A new set of results of the updated search is presented to the user to mark and re iterate the process if need be Thus with the same database and the same object each user may get di erent objects as the closest to the chosen one Figure illustrates the intent of the algorithm as well as some results ob tained by the system implementing it In Figure a an initial set of the top thirty similar objects is presented when a search for gures similar to man is conducted on a database of more than a VRML objects After the user marked a subset of the results as relevant or irrelevant and a couple of itera tions were run the nal set of results is produced and shown in Figure b The latter set of results can be contrasted with the initial set Not only more human gures were found among the top thirty but also their ranking among the top thirty has been improved Moreover gures which di er geometrically such as allien where the legs are open and man where one leg is folded and an arm is pointing forward were selected and re ect sematic similarity a Initial results b Final results Fig Searching the database for objects similar to man Three main issues are tackled in this paper The rst issue is the choice of features The goal is to have a representation which is compact su ces to uniquely identify each object and re ects similarities and dissimilarities be tween objects Such features can be considered as a signature associated with each three dimensional object just as a few keywords are associated with each document and are used when searching large databases or the web The second main issue is the choice of a distance measure which should not only re ect ob ject similarities dissimilarities well but should also be amenable to adaptations to enable re nement of searches The third issue is nding an adaptation rule on the distance measure We will show that the moments of the three dimensional objects surfaces up to some order are a relevant choice for features as only a handful of them are needed to represent the essence of the data We will also show that the weighted Euclidean distance leads to an e cient and e ective adaptation scheme based on the user s feedback Finally based on the user s input we nd the optimal weights for the weighted Euclidean metric in a way which re ects the user s desire and warps the space such that the relevant results are brought closer and the irrelevant results are pushed farther Using ideas based on Support Vector Machine learning algorithms SVM we formulate an optimization problem in which the weights are the unknowns and their optimal values correspond to the maximal margin between the distances The obtained optimization problem is convex QP thus ensuring a unique solution We view the contributions of this paper to be threefold First We propose a novel algorithm for retrieving three dimensional VRML objects an area which is relatively unexplored Second we propose to use moments as features rep resenting three dimensional objects and in particular we show how to use mo ments within a relevance feedback scheme While moments can have problems in pictures due to occlusions and self occlusions they can very well characterize three dimensional models which are given wholly Last but not least we show how to use relevance feedback in the context of SVM We tested the algorithm with a database containing over a objects given in VRML Simulations of the search mechanism exhibit very promising results With very few iterations usually the search process converges on what sample users had in mind for these searches The rest of this paper is organized as follows Section describes the object features we use Section describes the similarity measure and the actual it erative search process In Section we show some runs of the algorithm on a database of three dimensional VRML objects and explain the workings of the algorithm by way of demonstration Finally Section concludes the paper Object Features We assume an object is given in VRML i e it is a three dimensional object represented by a set of vertices and a set of polygonal faces embedded in three dimensions The features we choose to represent the objects are the moments computed for object surfaces assuming all objects are hollow For object D we denote the surface by D D s p q r th moment is the given by mpqr Z D xyzdxdydz The set of moments fmpqrg have a property of fundamental importance they uniquely determine and are uniquely determined by the object Thus the triple sequence of moments constitutes a full and complete object description and a partial object description can be obtained by using some subset of them Sampling to Approximate the Moments At the crux of our algorithm lies the computation of a subset of the p q r moments of each object which are used as the feature set Thus we perform a pre processing stage where the features are calculated for each database object A practical way to evaluate the integral de ning moments is to compute it analytically for each facet of the object and then sum over all the facets We use an alternative approach yielding an approximation of the moments The algorithm draws a sequence of points x y z distributed uniformly over the object s surface The number of points drawn from each of the object s facets is proportional to its relative surface area If we denote the list of points for a given object by fxi yi zig N i the p q r th moment is then approximated by mpqr N N X i x p i y q i z r i Normalizing the Objects We want the similarity measure to be invariant to spatial position scale and rotation of the di erent objects We therefore need to normalize the feature vectors of all objects The rst momentsm m andm represent the object s center of mass Thus the normalization starts by estimating the rst moments for each object represented as a set of surface sample points and subtracting them from each of these points i N xi yi zi T xi m yi m zi m T This amounts to positioning all objects so that their center of mass is at coordi nates thus removing any dependence on translation or spatial position This also sets each of m m and m to for all objects and thus renders them useless for further computations The second moments m m m m m and m repre sent the object s rotation and scale in the following manner When the second moments calculated for the object re centered at are ordered into a matrix M m m m m m m m m m and Singular Value Decomposition is performed the result may be written as U U SV D M where the unitary matrix U represents the rotation and the diagonal matrix represents the scale in each axis ordered in decreasing size The normalization continues with a second stage approximating the second moments for each object by computing them from the updated surface point data sets using equation into M After performing the SV D decomposition of the second moment matrix M we multiply each point by U to rotate the object back to a canonic position We also divide each point by to rescale the object so that its largest scale is To summarize each point is replaced by xi yi zi T U xi yi zi T Finally the algorithm should also determine each object s orientation rel ative to each axis To do this we count the number of points on each side of the center of the body In order to normalize such that all the objects have the same orientation we ip each object so that it is heavier on the positive side In counting the number of points and ipping according to it we are actually forcing the median center to be on a predetermined side relatively to the center of mass After applying all the normalization stages to each object the moments are computed once more up to the pre speci ed order As we indicated the normal ization xed m m m and m to and respectively for each and every object These are therefore no longer useful as object features Iterative Re nement Having two nite sets of moments in vectors X and Y constituting partial descriptions of database objects DX and DY respectively we can measure the distance between the objects using the square of the Euclidean distance d DX DY jjX Y jj Using the Euclidean distance alone the automatic search of the database will indeed produce objects that are geometrically close to the given one However these may not be what the human user had in mind when initiating the search Therefore we employ further parameterization of this distance by adding weights and a bias value d DX DY X Y W X Y b where W may be any matrix yet in the following we assume it is a diagonal matrix Given a set of search results a human user may consider some of them relevant and some of them irrelevant in spite of them all being geometrically close The adaptation of the distance function can be done by recalculating distances based on the user preferences The additional requirement is that the new distance between the given object and the relevant results be small and obviously the new distance between the given object and the irrelevant results needs to be large In essence this is a classi cation or a learning problem A way to formulate the requirements is by de ning weights on the components of the distance func tion and writing a set of constraints Denote the vector moments of the object for which the system is to search by O the vectors of moments of the relevant results by fGkg nG k and the vectors of moments of the irrelevant results by fBlg nB l The constraints posed on the weight function are then k nG d DO DGk O Gk W O Gk b l nB d DO DBl O Bl W O Bl b This creates a margin between the relevant and irrelevant results The above inequalities are linear with respect to the entries of W Denoting the main diagonal of W by we may rewrite the constraints as k nG d DO DGk O Gk b l nB d DO DBl O Bl b where the notation V for a vector V means multiplying each vector entry by itself An additional constraint is that the entries of W are all non negative Note though that we do not require b to be non negative and may therefore end up with a non metric similarity measure It can be shown see for example that the maximal margin of separation between the two sets of results is achieved by the with the smallest square of the norm min jj jj Choosing the with the smallest norm also renders the solution to the constraint system robust to the number of examples from each of the two subsets relevant and irrelevant and also the size of the rest of the database This is good when the above constraints are insu cient when nG nB M M being the arity of the feature vectors That is there are more unknowns than inequalities and therefore multiple possible solutions f bg all satisfying the constraints Thus at each re nement iteration we essentially need to solve the following for Minimize jj jj Subject to k nG d DO DGk O Gk b l nB d DO DBl O Bl b This Quadratic Optimization problem may be solved either directly or through the dual problem which proves easier when the number of constraints is much lower than the number of unknowns i e nG nB M The use of the bias in the formulation is crucial since it frees us from considering the boundary values and therefore choosing these values to be and does not lose generality Iterating The Search The system may use the new re ned distance function to perform a new search o ering the user a set of results to better suit personal preferences The user may on this new set of results mark preferences as was done for the previous search results The new relevant and irrelevant results sets may now be used to further re ne the distance function There is no limit by the system on the number of re nement iterations al lowed However our experiments showed that very few iterations were needed for any example before a human user is satis ed with the proposed search results Experimentation and Results We tested the algorithm with a database containing over a objects given in VRML First the database was pre processed All objects were sampled with points representing each object and then normalized as described in Section Further pre processing included computing a feature vector of the moments up to a pre speci ed order usually is su cient for each object At that point every VRML object has a small signature the vector of features associated with it Note that in the World Wide Web setup every signature need be calculated only once upon creation and then used as a key for all subsequent searches without ever having to access the VRML object itself At each test search we chose an object from the database and asked the system to produce the thirty closest objects These results are organized by their increasing distance from the desired object Given these we marked some as relevant and others as irrelevant according to what seems reasonable to a human viewer We then re iterated the search to take these preferences into account as described above in Section In Figure we show the Allosaurus an object to be searched In Figures we show the search results and their evolution from one iteration to the next based on the user s markings First Figure shows the results of the rst search iteration with unweighted Euclidean distance These results include four di nosaurs in addition to the one being searched and one of them was ranked as th We marked these four dinosaurs as relevant and four other objects as irrelevant Figure shows the best thirty results of the new search after the distance function had been weighted according to our preferences This added three more dinosaurs and they all ranked high The last two iterations brought in all the dinosaurs of the database ten including the original allosaurus and all of them are ranked at the top As can be seen from this example the search re sults improve with each iteration both in terms of the match of the results to the desired one and also in terms of the order in which the results are given re ect ing their closeness to the desired object Note again that the dinosaurs may di er geometrically Yet they are considered close because the user considered them semantically so Conclusions We propose a new algorithm for determining the content based similarity of three dimensional objects given in a standard representation such as VRML Three main issues are considered The rst is the selection of features that can represent the object in a compact manner and can thus be considered as a signature for the object similar to the way keywords representing documents The second issue is the selection of a distance measure that is adaptable and can enable learning in a way that takes semantics into account The third issue is he selection of an adaptation rule on the distance measure Using the normalized moments as features a weighted Euclidean function as the distance measure and the SVM algorithm to train the distance measure we introduce a novel learning mechanism that enables adaptation of the distance measure based on a user s preferences Experiments on a database of more than a objects exhibit very good results Not only did more semantically similar objects rank close to the query object from among those found to be geometrically similar but also objects which di er geometrically rank high An order of moments of only is su cient to represent the objects well Less than ve iterations need be run before the system converges to what the user had in mind The main aspects of this work can be extended Other features such as topo logical traits colors and textures can be considered as well We apply the basic search paradigm in one simple version namely let the search system present the user with results and allow the user to mark any results the user wishes as either relevant or irrelevant However one may think of other variants warranting experimentation Let the user mark only the results viewed as relevant and consider all other search results proposed by the system as irrelevant Impose further restriction requiring not only that all relevant results are close to the searched object but also that they be close to each other Impose yet further restrictions requiring in addition that each relevant result be far from each irrelevant result References IEEE Computer special issue on content based image retrieval Agarwal P K Sharir M and Toledo S Applications of parametric searching in geometric optimization Proc rd ACM SIAM Sympos Discrete Algorithms Alt H Behrends B and Bl omer J Approximate matching of polygonal shapes Proc th Annu ACM Sympos Comput Geom Alt H and Godau M Measuring the resemblance of polygonal curves th ACM Symposium on Computational Geometry Alt H and L J Guibas Resemblance of Geometric Objects In J R Sack and J Ur rutia editors Handbook of Computational Geometry North Holland Amsterdam Alt H Melhorn K Wagener H and Welzl E Congruence Similarity and symme tries of geometric objects Discrete Computational Geometry Vol Arkin E M Chew L P Huttenlocher D P Kedem K and Mitchell J S B An ef ciently computable metric for comparing polygonal shapes IEEE Trans Pattern Anal Mach Intell Arkin E M KedemK Mitchell J S B Sprinzak J andWermanM Matching points into pairwise disjoint noise regions combinatorial bounds and algorithms ORSA J Comput Atallah M J A matching problem in the plane J Comput Systems Sci Bertsekas D B Non Linear Programming Athena Scienti c Belmont Mas sachusetts Chew L P Dor D Efrat A and Kedem K Geometric pattern matching in d dimensional space Third European Symposium on Algorithms ESA Chew L P Goodrich M T Huttenlocher D P Kedem K Kleinberg J M and Kravets D Geometric pattern matching under Euclidean motion Comput Geom Theory Appl Chew L P and Kedem K Improvements on approximate pattern matching prob lems Third Scandinavian Workshop on Algorithm Theory Cox I J Miller M L Minka T P Papathomas T V and Yianilos P N The Bayesian Image Retrieval System PicHunter Theory Implementation and Psy chophysical Experiments IEEE Transactions on Image Processing Vol No January Cortes C and Vapnik V Support Vector Networks Machine Learning Vol No September M Duda R M and Hart P E Pattern Classi cation and Scene Analysis Wiley Hagedoorn M Overmars M and Veltkamp R C A Robust A ne Invariant Simi larity Measure Based on Visibility th European Workshop Comput Geom Huttenlocher D P and Kedem K Computing the minimum Hausdor distance for point sets under translation ISAAC Huttenlocher D P Kedem K and Kleinberg J On dynamic Voronoi diagrams the minimum Hausdor distance for point sets under Euclidean motion in the plane th ACM Symposium on Computational Geometry Huttenlocher D P Kedem K and Sharir M The upper envelope of Voronoi sur faces and its applications Discrete Computational Geometry Minka T P and Picard R W Interactive learning using a society of models Pattern Recognition O Rourke J and Toussaint G T Pattern Recognition In J O Rourke and J E Goodman editors Discrete and Computational Geometry pages CRC Press New York Paquet E and Rioux M A Content Based Serach Engine for VRML Databases Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Rui Y and Huang T S and Chang S F Image Retrieval Current Techniques Promising Directions and Open Issues Journal of Visual Communication and Image Representation pp Rui Y Huang T S and Mehrotta S Content based image retrieval with relevance feedback in MARS Proceedings of IEEE ICIP Schwarz J T and Sharir M Identi cation of partially obscured objects in two and three dimensions by matching of noisy characteristic curves Int J Robotics Re search Scholkopf B Support Vector Machines IEEE Intelligent Systems Vol No July August Sprinzak J and Werman M Exact Point Matching Proc Israeli Symposium on AI Vision and Pattern Recognition Elsevier Science Publishers Toussaint G T Computational Geometry and Computer Vision In B Melter A Rosenfeld and P Bhattacharya editors Vision Geometry pages Amer Math Soc Providence Vapnik V The Nature of Statistical Learning Theory Springer Verlag Ney York